Autonomous vehicles use various computing systems to aid in the autonomous control of a vehicle. Some autonomous vehicles may require some initial input or continuous input from an operator, such as a pilot, driver, or passenger. Other systems, for example autopilot systems, may be used only when the system has been engaged, which permits the operator to switch from a manual mode (where the operator exercises a high degree of control over the movement of the vehicle) to an autonomous mode (where the vehicle essentially drives itself) to modes that lie somewhere in between. However, conventional systems cannot render a smooth, yet safe transition between an autonomous driving mode and a driver-controlled driving mode. In particular, conventional systems cannot implicitly initiate a transition of control without the driver explicitly forcing control away from the autonomous vehicle control system.